Info-Mac 3/Info_Mac_1994-01.iso/Graphics/Utility/Sparkle 1.6/README 1st

Text File | 1993-08-04 | 30KB | 535 lines

Sparkle: A new mac MPEG player.
-------------------------------
Documents for version 1.5
This document looks best in 9 point monaco.

This document contains the following sections:
INTRODUCTION
ABOUT MPEG
ABOUT CONVERSION TO QUICKTIME
WHY DOESN'T SPARKLE DO....
SMALL THINGS YOU MIGHT NOT HAVE NOTICED
THE FUTURE
HOW CAN YOU HELP IMPROVE SPARKLE?

------------------------------------------------------------------------------
INTRODUCTION

Hi there, friendly users. This is release 1.6 of my mac MPEG player.
Version 1.0 of this code was based on the Berkeley MPEG unix code.
(Anyone who wants to play with the Berkeley code can get it from
toe.cs.berkeley.edu in pub/multimedia/mpeg.) It was released as soon as
it was usable. This release is a major rewrite in which I have tried to
address many of the problems with version 1.0.

I have completely rewritten about a third of the MPEG code, while
modifying another third. In particular the memory management, YUV
conversion, picture scheduling/ordering and error-handling have been
completely rewritten. These rewrites have allowed much more efficient
memory usage (a pure I-frame MPEG plays in about a sixth as much memory
as earlier, while a full IPB-frame MPEG plays in about half the memory.)
The new code also runs at 2 to 2.5 times as fast as the old code.

The user interface remains the same as in 1.0. It is ugly and
unsatisfactory, but it works and is not going to change for some time as
there are more important things I need to add to the code. However a
large number of little tweaks were added to the code. You will notice
that windows are updated more smoothly, multiple screen support is
better, the movie controller always changes its display to match the
state of the MPEG (playing, paused and such). Things like that.
Some of the changes I have made may seem gratuitous, or even for the
worse. Please be patient about them. These changes are in anticipation of
what I will add to the code, even though they seem like a step backward.

Please read all of this document before playing with the program. While
actually using the program is pretty simple, there are a few things you
should be aware of. Many of you may not care when I waffle on about
technical details. However I would ask all programmer readers (especially
people knowledgable about QuickTime, and especially Apple [and
ex-Apple :-( employees] to look at the tech sections and help me out with
comments and suggestions. Each time someone gives me a pointer on how to
do something it cuts a week or more off the release date of the next
version of Sparkle.

Features:
•    Standard mac interface with menus and windows.
•    Uses the QuickTime movie controller to control the MPEG viewing.
•    MultiFinder friendly, with good backgrounding behavior.
•    Saves MPEGs to QuickTime movies.
•    Can open multiple files at once.
•    Free.

To run it needs at least:
•    System 7.
•    QuickTime 1.5
•    A 68020 or better.
•    600K to play one 160x120 MPEG---to open more or large MPEGs increase the
    partition.

This program works fine, with good handling of errors, on my SE/30, but
that's the only machine I have to test it. If you find a bug that is not
caused by the various things listed below, please mail me with as many
details as possible, both about your machine and about what the  program
was last doing before it died on you.

I don't think there's much to say on the use of this program---you pretty
much run it like any other mac program. There is a section in this
document on tips that may not be obvious. When opening files, you can
choose to show all files, or only files with a .mpg suffix. If you choose
the "show all files" option and open some random file, don't be surprised
when you are told that that is not a valid MPEG file. If you set the
option to change file types, the file type of the MPEG file you are
changing will be changed to a Sparkle file, which will give it a nice
icon and allow you to open the file by double-clicking on it. The Finder
takes a bit of time to realise that you have changed the type of a file,
so the icon will not change right away. It will change when you open and
close the window, or reboot or stuff like that.

I have tested this program extensively under low-memory conditions when
it opens files and plays them. It should never crash under those
conditions. I have NOT done estensive testing of low-memory usage when
saving to QuickTime. So don't stress that too hard. In 600K you can
easily open, play and save to QT a 120x160 I-frame MPEG. In 1500K you can
open, play and ave to QT a 352x240 IBP-frame MPEG. Now in 1600K you can
open both a 352x240 and 120x160. You may even fit in two 120x160s. But I
can't promise that you'll be able to save to QT. The saving to QT may
cause a machine crash if everything goes wrong at once. I will some time
do a full trace through saving to QT like I did for opening files and
catch every possible low-memory condition I see.

Disk errors in various forms (bad sectors reading an MPEG file, no disk
space writing an QuickTime file, etc) will not crash, but the system will
put up an error alert and not handle the error very well (for example you
won't be given a chance to destroy old files to free up space on a disk).
Decent recovery from disk errors is on the list of things to do.

------------------------------------------------------------------------------
ABOUT MPEG

MPEG is an international standard for video compression. It compresses
frames at two levels. Firstly frames are compressed internally, and
secondly frame differences are compressed rather than transmitting full
frames.
To understand MPEG one should first understand JPEG. MPEG uses the same
ideas as JPEG for much of its compression, and suffers from the same
limitations.
JPEG compression begins by changing an image's color space from RGB
planes to YUV planes, where Y is the luminance (brightness of the image)
and U and V store color information about the image. Half the color
resolution in the horizontal and vertical planes is dropped, because the
eye is less sensitive to color than to brightness. These YUV planes are
then divided into 8 by 8 blocks of pixels. The 64 coefficients in this 8
by 8 block are fourier transformed concentrating the energy of the image
in a few coefficients at low frequencies. The high frequency terms of the
fourier transform can be discarded as the eye is not sensitive to them.
The resultant fourier transform coefficients are then encoded using a
variable length coding scheme (basically Huffman coding) so that
frequently occuring patterns of coefficients are transmitted in few bits
and rare patterns transmitted in many bits.

MPEG goes beyond this by adding support for inter-frame compression. This
compression works by realizing that most video consists of foreground
objects moving over a largely static background. Thus rather than
transmit the foreground and background pictures over and over again, what
is transmitted is the motion of the foreground objects. Note that this is
different from the way QuickTime does interframe compression. What
QuickTime does is just to subtract the two images from each other and
compress the resultant image (which is hopefully largely blank space.)
The MPEG scheme gives much better compression, but is much harder to
program. It is essentially a pattern recognition problem, looking at a
set of frames and deciding what pixels in the frame correspond to moving
objects---the sort of thing humans are very good at and computers very
bad at. For this reason, a complete MPEG compressor is complex and very
slow.

MPEG movies consist of three types of frames. I-frames are like JPEG
images and are compressed by themselves. P-frames are compressed based on
motion relative to a previous frame. B-frames are compressed based on
motion relative to both a previous frame AND a future frame. How do you
know what the future frame is? Well the MPEG data is not stored in the
same order as it is displayed. You have to decode future frames before
the B-frames on which they depend, then buffer the future frame
somewhere. This is why MPEG players need rather more memory than
QuickTime players.
An MPEG movie can consist of only I-frames. This will be far from
optimally compressed, but is much easier to encode because the pattern
recognition is not needed. Such a movie is pretty much what you would get
if you made a QuickTime movie and used the JPEG codec as the compression
option. Because the I-frame movie is so much easier to calculate, it is
much more common. Sparkle checks if a movie uses only I-frames and if so
reduces its memory requirements since such movies do not need complex
buffering. In the PC world, many people talk about XING type MPEGs which
are pure I-frame MPEGs. These are produced by XING hardware on PCs and
played back using the XING MPEG player.

One problem with the MPEG standard is that many vendors seem to feel
which parts of it they support are optional. XING, for example, often
does not ends its MPEGs properly. It does not start frame numbering
properly, and does not correct frame numbering after MPEGs are edited.
GC technologies produces MPEGs that have the frames essentially random
numbered, and has garbage frames at the start of its MPEGs.
Wherever possible I have tried to adapt my code to common pathologies in
MPEG encoders.
I have also built in powerful yet computationally cheap
error-detection and recovery. For example a recent MPEG posted to usenet
drew widespread complaints because some of the uuencoded text was garbled
and the resultant MPEG crashed pretty much every decoder out there. But
Sparkle noticed the error and went on quite happily. Sparkle has also
proved quite robust in the face of MPEGs I have deliberately corrupted.
If you come across any MPEG file that causes Sparkle to crash or produce
garbage, I WANT TO KNOW ABOUT IT. With a copy of the file, I can trace
through Sparkle, find just what causes the crash, and make Sparkle even
more robust.

For more details on MPEG, read the MPEG FAQ on USENET. It is posted once
a week to the picture groups and to news.answers.

------------------------------------------------------------------------------
ABOUT CONVERSION TO QUICKTIME

The following are notes I've made on conversion to QuickTime. I have
investigated this issue extensively, but not exhaustively. If someone has
comments on the subject---more extensive notes than I have, corrections,
whatever, please tell me.

All times I give are on my SE/30 with a 32-bit screen. People should
extrapolate to their machines---I guess LC IIs are about half as
fast and Centris/Quadras three to six times as fast.

The useful codecs are video, cinepak (used to be compact video) and jpeg.
JPEG compression at normal quality gives files of very good quality and not
much larger than pure I-frame MPEGs. A 120x160 image can play back at about
4fps. Translated to an 040 and you get a useful frame rate. However JPEG
has a major problem in that when it decodes to a 32bit screen, it draws
directly to the screen, not to an offscreen Gworld unlike other codecs.
This produces movies with obvious tearing artifacts. When fast-dithering is
used to draw to other screen depths, it works fine. I don't understand why
this problem with 32 bit screens should be the case, but I have told Apple
about this problem and maybe it'll be fixed in a later release of
QuickTime. Meanwhile write to Apple and complain---they are holding back a
useful capability.

With the video and cinepak compressors, it is very important to check the
key-frame rate checkbox. Key-frames are like MPEG I-frames. They are
compresed standalone and do not depend on other frames. The other frames
produced by the movie codecs depend on previous frames. Setting the
key-frame rate guarantees that at least that rate of key-frames (one
frame in used. for example) will be used. Checking the key-frame rate
checkbox allows the movie to use intra-frame compression (ie not just
key-frames) and gives movies half as small as they would otherwise be.
The lower you set the key frame rate to (this means a larger number in
the QuickTime saving options dialog box) , the smaller you movie will be.
For example a 72K MPEG (48 frames, 120x160, pure I-frame) became a 290K
movie without keyframes, a 160K movie with a key-frame rate of 1 in 8,
and a 138K movie with a key-frame rate of 1 in 96.
The price you pay for a low key-frame rate is that the movie has more
difficulty when playing backwards, or when randomly jumping around. I
don't find it a problem and usually use a key-frame rate of about 1 in
100, but try for yourself to see what things are like.
Video gives better quality results when a higher key-frame rate is used.
Strangely cinepak appeared to give lower quality results (as well as a
larger movie) when more key-frame were used.
I'll have to investigate this further---I may have become confused when I
was making the measurements. Anyone want to confirm or deny this?
(For comparison, this same movie became a 90K JPEG movie.)

I find video and cinepak give much the same file sizes at the same
(around normal) quality setting. The cinepak file is consistently a
little larger, but not enough to matter. The video file is consistently
lower quality for the same size as the cinepak file. However the video
low quality artifacts (blocks of solid color) I find less psychologically
irritating that the cinpeak low quality artifacts (general fuzzing of
borders like everything is drawn in crayon and watercolor).
However cinepak has the advantage of playing back much faster than video.
For a 120x160 image on my 32bit screen, I can get smooth playback with
cinepak at 24fps. Video can do smooth playback up to about 16 fps.

Fast dithering seems to be a good job for speed (at the cost of quality).
Unlike earlier versions of QuickTime, with 1.6.1 I found the same speed
of playback (ie same degree of skipping frames or not) at every screen
depth but 2 bit depth.

Cinepak can support a largish MPEG to QuickTime movie (352x240) at 6fps
on my mac, but no faster.

Compression using cinepak is SLOW SLOW SLOW. A 120x160 frame takes about
10 seconds to compress. A 352x240 frame takes about a minute. In this
time your mac is stuck---it looks like it has crashed. Don't start saving
to cinepak QuickTime unless you are prepared to walk away from your mac
and not touch it until it's done.
QuickTime 1.5 did not include anyway to do this compression in small
chunks so that it would run nicely in the background. I received word
today that QuickTime 1.6 does have this capability, so once I get the
relevant techincal documents and read them, I will add this ability.

See the WHY DOESN"T SPARKLE DO... section for information about MPEG
frame rates and their relationship to QuickTime frame rates.

------------------------------------------------------------------------------
WHY DOESN"T SPARKLE DO...

•    Why doesn't Sparkle tell you what rate an MPEG should play at so you
    can use that rate for the QuickTime movie?
In my opinion the single stupidest thing the MPEG committee did was the
way they defined frame rates. Rather than allowing an MPEG file to be
whatever rate is wanted, like a QuickTime file, only one of eight fixed
rates is allowed. The minimum of these eight rates is 24fps. This is
clearly insane as many people don't have the hardware or the need to
display at this rate. Many MPEGs are being produced at 16 or 12fps. These
MPEGs set the frame rate to a bogus value cause they can't do anything
else. In my experience, most 120x160 MPEGs work best at about 16fps.

Now some MPEGs do give a display rate that is valid. However these are
usually large MPEGs, and their frame rate is 30fps. So unless you have a
quadra with extra hardware quicktime just won't be able to play the movie
at that rate anyway.

Because of these, for now I don't bother to point out the frame rate of
the MPEG. This will certainly change as I improve the program. Even if
you don't set that frame rate to the QT frame rate, it's interesting to
know.

Also note that some MPEGs are converted video of what was originally a
movie. Now movies run at 24fps, but video is 30 fps. So every five frames
a movie frame is doubled giving an extra video frame. This works fine
when playing at 30 fps but gives a strange jerkiness when playing at 6fps
or so. A future Sperkle enhancement will seek out these duplicate frames
and eleminate them, but for now you are stuck with them.

I know it is a pain to creat a movie at one frame rate and find it sucks,
so you have to recreate it at a second frame rate. It would be nice just
to use the movie you have but change the frame rate from say 16fps to 12
fps. This will be added to a later version of Sparkle.

•    Why doesn't Sparkle read .gl, .dl and .fli files?
One reason is that those file formats are awful. They give these dinky
low contrast horribly dithered images no-one would want to look at, and
they usually only have about 10 frames. Maybe, years from now, I'll add
those conversions, but they're about as low a priority as you get.

•    Why doesn't Sparkle convert QuickTime to MPEG?
It will. That will be the major new feature of version 2.0. I have
recently released code from Stanford that does this. The word from the
DOS ports of it is that it is slow as molasses, that it makes cinepak
look like greased lightning. Anyway I will see what I can do about adding
this to my code and getting it fast enough to be usable. This is a large
task, so I can't give a date. Maybe you'll see a working (though not
speed-optimized) program two or three months from now.

•    What about audio?
MPEG has audio compression as well as video compression. I do not really
understand how the audio compression works. That is, I have the MPEG
standard that explains the bit patterns used and such, but I don't
understand the overall ideas behind the encoding. I don't think I can do
a good job in terms of high speed, good quality, low memory usage and
such, unless I have such an understanding.
Does anyone out there have any references on how the MPEG psycho-acoustic
coding works? Technical journals, books, elementary or advanced
treatments. I don't care what you suggest---anything is better than my
present zero knowledge.

•    What about Windows for Video?
I would be nice to support .AVI files. But right now I know nothing about
.AVI beyond the fact that it exists. Again any info anyone has is
appreciated. Until I know how .AVI works, how it fits into the windows
environment etc, I can't even tell you if it's practical for Sparkle to
try to support .AVI, let alone start the necessary coding.

•    Why don't you write an MPEG codec?
It is not yet clear to me that an MPEG codec has any particular use.
As far as the MPEG algorithm goes, my code is not yet fast enough to give
real video, even on quadras. For a 120x160 movie I get 1 fps on my SE/30.
While I hope to get at least twice that as I continue to revise the code,
I don't think I can get much beyond it. The best I could ever get if I
wrote in pure assembly is 4 fps (the JPEG codec rate) which isn't really
good enough.
Next the MPEG file structure is rather different from the QuickTime file
structure in the way that different pieces of data are tagged and
synchronised. Things would not be like the JPEG codec where a program can
slap a short header onto a JPEG file, feed it to the JPEG codec and have
a usable picture. The program would still have to know a lot about the
MPEG file structure to use the codec, so the whole point of the codec as
being usable by programs that don't know the compression algorithm
details is lost.
Also I suspect the way MPEG uses forward image prediction in B-frames
will cause QuickTime problems both with buffer allocation (where will the
future predictive frames be stored) and in random access.

So for now I think a separate MPEG<->QT program is most useful. Of course
when Sparkle is maxed out as an application (and when QuickTime has
evolved more) I may reconsider the issue, but it won't happen soon.

•    Why is Sparkle so slow?
Partially because it's doing lots of work. Partially because parts of the
Berkeley code still need to be rewritten by me.

There are now two main bottlenecks. Parsing and the IDCT (inverse
discrete cosine transform). Parsing should speed up a bit when I
replace the present (really simple) disk buffering subsystem with a much
more sophisticated random-access buffering scheme that eliminates the
redundancies that plague the present parsing.
The IDCT code is in C. Writing it in assembly would give a fair speed
boost, but is a fair bit of work. Can anyone (an Apple employee) tell me
how to access the IDCT code in the JPEG codec. If I could call that code,
I could easily give Sparkle a speed boost.
Additionally MPEG defines some really finicky details about how to
reconstruct the image once the data has been parsed. I'm not sure we need
to be that finciky, and I suspect I can drop some of the processing
that's done at negligible cost in image quality. Anyway I have to
experiment with that.

Right now I find Sparkle fast enough to be usable, so don't expect
further speed increases until after new features (MPEG encoding and
sound) are added.

•    Why does the display of frame numbers sometimes jump backwards and
    forwards?
This display has a somewhat schizoid purpose. One role is to tell the
viewer what the frame number of the current frame is. A second role is to
provide visual feedback that something is happening when a frame is being
decompressed. Now when B-frames are decompressed, sometimes two frames
have to be decompressed before anything is drawn on the screen (the
future frame must be decompressed, and then the B-frame). But if I make
the text display climb through two numbers not one, it looks like random
frames are being lost. So what you have now is a bareable kludge that
serves both masters. I have an idea for how to deal with this problem
better, by not trying to display these two different pieces of
information in the same way, but those will wait until new functionality
is added to the code.

•    Why does the MPEG sometimes stop before the highest frame number?
This can happen for two reasons. One is that some MPEGs (like those
produced by GC technologies) have two garbage frames at the start of the
file that cannot be decoded and are skipped by Sparkle. Right now Sparkle
isn't clever enough when it counts the nmuber of frames to notice that
these frames are garbage and should not be included in the count of total
frames. In future this smartness will be added.
A second reason is that if a frame is corrupt, it may be displayed as an
image with some junk, or it may be skipped entirely because Sparkle just
can't deal with whatever error corrupted the file.

•    Will I make the source available?
Yes, at some point. Right now it is horrible, being as it is a hybrid of
my code and the Berkeley code. I have very fixed ideas about C/C++ style,
which are very different from the ideas of Berkeley. I doubt anyone could
extract anything useful from the present mess.
However every two weeks or so, once my latest experimentation and ideas
are resolved, I go through a few files and clean them up, fix up things I
commented as quick kludges, etc etc. So the code is evolving to a
publicly acceptable form.
I am also passing on to Berkeley my ideas for speeding up the code and
making it more robust to corrupt data, so with luck MPEG players on other
platforms will also improve with time.

------------------------------------------------------------------------------
SMALL THINGS YOU MIGHT NOT HAVE NOTICED

The movie controller is not as functional as a real movie controller.
This is because I have not yet replaced the Berkeley disk buffering
sub-system with my smart, random-access sub-system.
So the movie controller gives you play, pause, forward one frame and
rewind, that's all. Rewind is given, like on a normal movie controller,
by clicking on the backward-one-frame button with the option key down.
You can also rewind once the end of a movie is reached by clicking the
play button twice.

The visual clue that the movie is being converted to QuickTime is that
the movie controller loses its steppers. This is not a particularly
obvious fact and will at some point be changed. For now it works once you
realize this fact.

When a movie is being converted to QT, you can either stop the conversion
or pause it. To stop the conversion, use command-S (or the equivalent
menu option.) This will save the movie using the frames created so far.
To pause the conversion use command-P, or the equivalent menu option, or
click on the movie controller's play/pause button. The quicktime
conversion will stop until you start that movie playing again. This is
occasionally useful if you want to pause a cinepak conversion to do a
short job on your mac.

If you are partway through a movie and save to QT, the movie will be
rewound to the beginning for you. You do not need to be at the start of a
movie to save to QT.

You can set the temporal quality options when saving to QuickTime
separately from the spatial quality options. If you need to do this, hold
down the option key while using the quality slider, and it will become a
temporal quality slider. (This may be a new feature of QT1.6, and may not
work in QT1.5.)

The code runs fastest on a 24bit screen because it then does not need to
dither the image to the display. The dithering adds a 15% or so slowdown
to the code. So if your screen has a 24bit mode, use it.
My screen does not do 16 bits. I have some ideas about things to do that
would improve playback on a 16bit screen, but cannot test them. If some
programmer type out there with experience in QuickDraw, QuickTime,
dithering, Gworlds, that sort of thing wants to contact me, I can mention
my ideas for how to support a 16bit screen better and try out sample code.

------------------------------------------------------------------------------
THE FUTURE

The basic outline for now is
    2.0 Opens and play QT movies. Converts them to MPEG.
    2.5 Cleaned up faster, smaller version of 2.0.
    3.0 Handles sound.
    3.5 Cleaned up faster, smaller version of 3.0.
While adding these large additions I'll fix small things as I go, and as
I have time. I don't see the user interface improving much for some
time---more important things need my time.

In mid August I will be going to New Zealand and looking for a job.
Depending on how job hunting goes, and what job I find, the time I can
devote to this code may shrink, but I'll do what I can when I can.
Anyone want to hire me? I also have UNIX/X-windows programming and system
management experience, and a MS in physics.

------------------------------------------------------------------------------
HOW CAN YOU HELP IMPROVE SPARKLE?

•    Any info on psycho-acoustic encoding?
•    Any info on MicroSoft Video for Windows?
•    Assembly code for IDCT, or a way to access the JPEG codec's IDCT code.
•    Why is QuickTime so slow about creating text track movies? If you have
    code that creates text track movies fast, please show it to me.
•    Any bug reports.
•    Any ideas you have or suggestions. Your suggestions may go onto the
    list of things to do (currently two pages of single line items) but will
    probably be acted upon at some point. People have suggested several
    things to me I would not have thought of myself, so I do want your
    feedback.
•    Anything on the QT16 CD ROM beyond the technote at ftp.apple.com.
    I have full docs for QT15, but already I've learned of one aspect of 1.6
    that I coul make good use of. There may be others.
•    Tell me if you have a graphics card/screen that does both 24 bits and
    16 bits, and if you know some QuickDraw and QuickTime concepts. I need
    some feedback on ideas I have for 16 bit screens.
•    How can I immediately tell the Finder that I have changed the
    attributes of a file (ie changed file type, added a custom icon,
    whatever)?
------------------------------------------------------------------------------
THANKS

    Many people all over the internet have helped me write this code.
    Thanks to the people at Berkeley. Even though I'm gradually destroying
their code as I replace it with my own, they helped get this project
started.
    Thanks to the people at Stanford whose code I hope to start using real
soon now.
    Thanks to various usenet personalities who answered mac programming
questions, mailed me quicktime header files and such. Special thanks to
ldo in New Zealand, and Jon W{tte in Sweden, and bryanw, keeper of the
MPEG FAQ, who mailed me about the Stanford MPEG encoder.
    A special individual thank you goes to DS (he didn't want me to give his
name but he knows who he is.) DS mailed me a CD ROM and ten floppies of
information about QuickTime components after I complained that Apple was
not making this information easily available. Already that knowledge has
helped this version of Sparkle handle update events much better---no more
flashes of images. Once I add random access to the code, his help will
really come into its own.
    Thanks to Apple for making the greatest computers in the world, and to
Symantec for creating such a great programming environment.

Maynard Handley
[e-mail]
Aug 04 1993